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Abstract. Nowadays, television programmes are not only accessed through a 
conventional TV set; they can be viewed through streaming services on the internet, 
smartphones, and tablets to name but a few media. For language learners, this 
development has opened up opportunities for accessing authentic materials in foreign 
languages outside of the classroom. Studies have shown that the simultaneous 
presentation of aural and visual information in captioned audiovisuals in a foreign 
language has the potential to improve language learners’ listening skills (Mitterer 
& McQueen, 2009). Recent research has prompted extension of this discussion to 
online language learning behaviour and to analyses of what information learners 
attend to while watching captioned programmes (Vanderplank, 2015). This doctoral 
research seeks to contribute to this new perspective and explores actual viewing 
behaviour of native speakers and language learners of Japanese while watching a 
captioned Japanese variety show through the use of eye-tracking and questionnaires. 
This paper discusses the context and methodology of this study as the data analysis 
is in its initial phase. It is hoped that the results will deepen our understanding of 
language learners’ reception of Japanese captions and make suggestions as to how 
such authentic material can complement Japanese language learning. 
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1. Introduction 

Learning materials that contain Audiovisual (AV) input, such as television 
programmes, have been a topic of particular discussion in research that explores the 
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use of leisure-oriented media products in Foreign Language (FL) learning. Studies 
have shown that the simultaneous presentation of aural and visual information in 
an FL through the use of same-language text on screen has the potential to improve 
language learners’ listening skills and ability to segment speech, recognise word 
boundaries, and retune perceptual processing (Charles & Trenkic, 2015; Mitterer & 
McQueen, 2009). This shows the potential of such same-language text as an aid to 
learners for visualising FL aural input. Building on studies that examined the effects 
of same-language text on screen, recent research has increasingly focussed on what 
language learners do and which information learners attend to while watching 
videos with such text (Vanderplank, 2015). Additionally, recent CALL research 
(O’Rourke et ah, 2016) has analysed online learning behaviour in Synchronous 
Computer-Mediated Communication (SCMC) through the application of eye¬ 
tracking technology and principles in Second Language Acquisition (SLA). 

This doctoral study is no exception to this trend, in that the focus is on the reception 
of authentic AV material. However, it expands the discussion to online viewing 
behaviour of Japanese Language Learners (JLLs) while watching a Japanese 
variety show with same-language text on screen. This study is part of a larger 
project run at Dublin City University (DCU) that focuses on the reception of this 
particular media product by different viewer groups. What makes Japanese variety 
shows different from other types of captioned authentic material studied in SLA 
research is the integrated text on screen. Rather than incorporating white text 
that covers the imagery as little as possible, it is common to see same-language 
text placed on almost half of the television screen in bright colours, a variety of 
typefaces, and accompanied with different sound effects. A good grasp of how 
such material is received by JLLs helps to effectively complement the Japanese 
language classroom with its use. This paper discusses the methodology of this 
doctoral research as the data analysis is still underway. 


2. Context of this study 

2.1. Main objectives 

Two main objectives form the basis of this research. The general overarching aim 
of this study is to gain insights into focal attention and online viewing behaviour 
of JLLs while watching a Japanese variety show with same-language text on 
screen. A second, more specific goal is to investigate the visual attentional shifts 
of JLLs against the audio track of the media product in order to gain insights into 
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vision-sound matching or speech monitoring practices of JLLs. It is expected that 
the findings of this study will contribute towards a better understanding of the 
subconscious language learning process that precedes the improvement of FL 
listening skills as has been observed in research on the effects of captions (Charles 
& Trenkic, 2015; Mitterer & McQueen, 2009). 

2.2. Participants and data collection 

Data has been collected from 45 JLLs who were at different stages of their Japanese 
language study at DCU at the time of data collection (three first-years, 15 second- 
years, 20 third-years, and seven fourth-year students). Data has also been gathered 
from seven native Japanese speakers who function as a reference group with which 
learner behaviour is compared. The eye-tracking experiments were conducted in 
an experimental setting in both Ireland and Japan with a portable Tobii X2-60 eye- 
tracker attached to a laptop. 


3. Methodology 

3.1. Methods 

A Mixed Methods Research (MMR) design has been developed in order to 
address the objectives of this study as well as possible. Whereas the first aim 
requires a largely quantitative approach for searching trends in focal attention and 
online viewing behaviour of JLLs; the second objective calls for the researcher 
to zoom in on individual viewing experiences of participants in order to uncover 
how JLLs monitor speech and attend different types of input. The abundance of 
highly contextualised information emitted from television programmes gives FL 
learners the opportunity to attend or select, whether consciously or subconsciously, 
input that fits their capabilities and supplements their personal learning journeys 
(Gilmore, 2007). Quantitative and qualitative approaches combined can therefore 
help explore such processes more thoroughly and inform a better judgement of the 
appropriate use of captioned television programmes by JLLs. 

This research collated data gathered from eye-tracking technology and 
questionnaires. The data generated by eye movements allows for both a quantitative 
and qualitative assessment of focal attention based on numerical fixation data within 
particular Areas Of Interest (AOIs) in the video stimulus and by overlaying the video 
stimulus with a replay of the gaze of participants. As research on same-language 
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textual inserts in screen-based media have shown that such text on screen fosters 
learners’ listening skills, data on eye movements is approached with a multimodal 
lens. AOIs are defined based on the type of resemblance identified between the 
textual inserts and corresponding dialogue (e.g. identical, similar or different 
representations). The questionnaire responses contain information on factors that 
may have influenced participants’ focal attention such as demographics, experience 
with Japanese television, and opinions on the text on screen. Furthermore, during 
the debriefing at the end of each experiment session, participants provided some 
additional insights into their thoughts on the experiment. 

3.2. Data analysis phases 

For the purposes of this study it was decided to divide the data analysis into three 
phases. These phases are defined as follows: 

• Identification of resemblance types in the video stimulus 

• Generation of numerical eye movement data for each resemblance type 
and selection of a select number of gaze replay segments 

• Interpretational analysis of gaze replay segments 

A nested model is applied in order to facilitate the shift in focus for the eye-tracking 
data analysis from a quantitative into a qualitative approach. This means that a 
smaller, representative sample is further explored for the qualitative analysis. 
Although this overview primarily describes how the eye-tracking data is analysed, 
field notes and questionnaire responses are consulted throughout this process. 


4. Potential implications of findings and conclusion 

FL learning through authentic material generally emphasises instruction that is not 
focused on the linguistic aspects of a language; rather, it is based on language in 
use and often accompanied with awareness-raising tasks (Gilmore, 2007). This 
implies different teaching formats, instruction styles, and consideration for ways 
in which technology can enhance the language classroom. Although this doctoral 
research is foremost a study of language learning behaviour, insights into focal 
attention and feedback from JLLs on the use of captioned AV material at different 
academic stages can help deepen our understanding of how to raise Japanese 
language awareness and the appropriate time to use this type of language learning 
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material. It is expected that the findings of this study will show that different types 
of resemblance between same-language text on screen and dialogue is accompanied 
with different speech monitoring practices. 
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